One of the goals of the PASI project is to have a shared view of student data across the province. In order to accomplish this goal some mechanism is required in order to enable PASI clients (school and school authorities) to be made aware of and synchronize with the most current data available in the PASI Core.

A key non-functional requirement for the data synchronization capability is that PASI clients must be made aware of changes to data in the PASI Core near real-time basis. A key challenge to meeting this requirement is that the PASI Core has no way of notifying PASI Clients directly (the PASI Core is a set of services that PASI Clients call - not the other way around). Another challenge is determining which PASI Clients are actively interested in being made aware of updates in near-real time (if a PASI Client is not currently online the Core should not expend effort attempting to notify the client.)

With these two key drivers in mind the PASI Core decided to use a "long-polling" approach. This approach is similar to the long-polling approach use in the Bayeux Protocol[1] enabling web browsers to receive messages from the server through a client initialed request.

Long-polling described by the Bayeux Protocol is as follows:

"Long-polling is a polling transport that attempts to minimize both latency in server-client message delivery, and the processing/network resources required for the connection. In "traditional" polling, servers send and close responses to requests immediately, even when there are no events to deliver, and worst-case latency is the polling delay between each client request. Long-polling server implementations attempt to hold open each request until there are events to deliver; the goal is to always have a pending request available to use for delivering events as they occur, thereby minimizing the latency in message delivery. Increased server load and resource starvation are addressed by using the reconnect and interval advice fields to throttle clients, which in the worst-case degenerate to traditional polling behavior."

The Data Synchronization services allow a PASI client (client) to call the PASI Core (core) and identify if any data is newer in the core than on the client. If there is newer data in the core the client can call a specific service to retrieve the newer data.

If there is no new data immediately available for the PASI client they may stay connected to the IsDataAvailable service for a specific amount of time. This is where the long polling technique described in the previous section is used.

Figure 1: The above diagram shows an example of the service flow.

1. The PASI Client calls the PASI Core to check if new data is available.
2. If there is new data available the core would have returned the Ids of the new data. The PASI client will then call the PASI Core service operation to get the new data. In this case new student data was available so the GetStudent service operation is called.
3. The PASI Client calls IsDataAvailable starting the process again so a connection is maintained.

In order to identify what data is different on the PASI client (client) than in the PASI core (core) a couple strategies are used.

Strategy 1

The first strategy used is to track a PASI Core version number both on the client and core. Each data contract that can be synchronized with the PASI core has a PASICoreVersion property on it. The PASICoreVersion property is a numerical representation of the LastUpdateRowVersion column which is on each table in the PASI database[2]. This number is unique across the entire PASI database allowing easy identification of what has changed since a particular version number. The number is set by the core on creation of a record and increased every time the data is modified.

The PASI client is responsible for maintaining a copy of the PASICoreVersion value so it knows its current version. The client will also pass the maximum known PASICoreVersion per type to the IsDataAvailable service operation so the core knows the latest version the client is aware of.

New data is identified by the client passing its data version number to the core when checking for new data. If the client has never received data from the core this number would be zero. If the client has previously received data from the core the version number passed back should be the maximum version received on the data returned.

For example:

If the client received a student with a version of 1 and an enrolment record of version 2, 1 would be passed back as the maximum PASICoreVersion number for Student and 2 for StudentSchoolEnrolment.

The client must:

1. Identify what data types they wish to be notified on (Student, StudentStatus, StudentSchoolEnrolment, etc.)
2. Call IsDataAvailable
3. Call the appropriate data method(s) to retrieve the updated data (GetStudent, GetStudentStatus, GetStudentEnrolment, etc).

Strategy 2

In order to handle the scenario where a client is potentially missing data that it should have (or have data the core isn't aware of) a second strategy is used to identify differences between the client and the core. With this optional approach the client sends with its request a MD5 hash of the all object versions a client has of a specific data notification type (i.e. loop through all the students and create a hash of the object version). This hash value is sent to the core and it can determine if the values the client has are the same as the core. If these hash values differ (and the MaxPASICoreVersion match) then the client is out of sync with the core. In this scenario the core will return a 'full list'* of Id and PASICoreVersions for the requested type. The client can then compare each Id and PASICoreVersion to the client's local list and determine what record(s) is out of sync

* NOTE: The IsDataAvailable service has a maximum number of key/version pairs it will return on a single call. If the entity being synchronized exceeds this limit the IsDataAvailable service must be called multiple times to retrieve all key/version pairs in the event of a hash mistmach. For example: if your system has 250,000 Course Enrolments and you receive a ResyncRequired status when performaing a hash check the first 150,000 key/versions will be returned with the ResyncRequired status. The version of the last record will be used to called IsDataAvailable again to retrieve the next 100,000 key/version pairs before a record by record compare can be completed.

Calling IsDataAvailable and setting the WaitTimeInMinutes parameter to zero will check to see if there is new data that the PASI client is interested in and return a result indicating if new data is available immediately.

Calling IsDataAvailable and setting the WaitTimeInMinutes parameter to something greater than zero will check to see if there is new data that the client is interested in and return immediately if there is new data available the PASI client is interested in. If there isn't new data available this method will wait for one of two events to occur:

1. Data is updated in the PASI core. At this point another check is preformed to see if there is new data available the PASI client is interested in.
2. The timeout period the PASI client specified in minutes expired.

Once completed the PASI client will call IsDataAvailable again to continue the long polling process.

In either the immediate or the delayed response the PASI Core handles the service call through asynchronous WCF service call. An example of this approach is described here: How to: Call WCF Service Operations Asynchronously

When the PASI core does respond the IsDataAvailable service can be called immediately again to minimize the amount of time before a PASI client is notified of data changes.

The goal of the IsDataAvailable service is to have clients continually connected using the WaitTimeInMinutes parameter to determine how long this connection will be maintained, thus reducing latency between update notifications. A longer wait time means less service calls and less network traffic.

The primary reason for having the client provide the value for WaitTimeInMinutes on IsDataAvailable is driven by the unpredictability of intervening network conditions between the PASI Client and the PASI Core. This unpredictability will require that there will not be a single value for WaitTimeInMinutes that works for all clients (even if the clients are all using the same vendor SIS software.)

PASI recommends the following:

1. Initially set the WaitTimeInMinutes value to 1 minute and once confirmed the service is working as expected in your environment then trying to increase the value to decrease the number of services calls when calling.
2. An open / connect timeout should be configured in the client to identify when connectivity to PASI is failing due to an absolute inability to connect. This would normally be associated with network outages somewhere between the PASI client and the PASI core. This requirement is unrelated to the choice of WaitTimeInMinutes, but mentioned here to ensure that the issue of simply not being able to connect to PASI Core at all is dealt with as a separate issue.
3. A read / receive timeout should be configured in the client communication stack (socket read timeout in Java, WCF timeout in .NET). This timeout value should be slightly larger than the WaitTimeInMinutes value being passed to IsDataAvailable (i.e. 30 seconds longer than WaitTimeInMinutes to allow some time to process the request). Once the WaitTimeInMinutes has passed and the PASI client has not received a response from PASI the client can be confident they will never receive a response on this request. So if WaitTimeInMinutes is well chosen, the client’s read/receive timeout should normally not occur. If it does occur it will indicate that the chosen value for WaitTimeInMinutes is incompatible with the intermediate network between the PASI Client and the PASI Core. This timeout is what will provide the PASI Client the necessary feedback it needs regarding its choice of WaitTimeInMinutes.
4. When a read / receive timeout occurs, the PASI Client should log the timeout event in an operational log for system operators to review. Ideally, if the timeout occurs repeatedly, the PASI client would use this information to automatically decrease the value it uses for WaitTimeInMinutes (ensuring it does not go below one minute) At minimum, though, the PASI client should log the timeout error so that operational staff who are managing the SIS environment can reduce the value for WaitTimeInMinutes appropriately.

On each 'Get' service (e.g. GetStudent, GetStudentSchoolEnrolment, GetSection, etc.) request data contract there is a field called ExpectedVersion. This field should be populated with the version numbers returned from IsDataAvailable which allows PASI to only return objects the client is expecting.

When the 'Get' service compares the ExpectedVersion to the version currently in PASI. If they match the requested object is returned. If they don't match the object is not returned an processing of the service call is stopped. For example if 10 Id/Version pairs were requested from a 'Get' service and upon reaching the 5th record the ExpectedVersion didn't match PASI's current version the first 4 objects would be returned but the remaining 6 objects would not.

If the ExpectedVersion didn't match it means that PASI has either a higher or lower version than requested. A higher version is possible because between the time IsDataAvailable was called and the 'Get' service was called the object was modified again. PASI having a lower version than the ExpectedVersion is a more rare edge case. PASI services run on multiple servers. In this scenario the caller must have called IsDataAvaile on one server and the 'Get' service on another that has not yet learned about this most recent change.

If the ExpectedVersion field is used the highest version returned from the 'Get' service must be the version passed into the next call to IsDataAvailable This ext call to IsDataAvailable must happen after the 'Get' calls are completed to have the correct version number. If a PASI client relied on the maximum version coming back from IsDataAvailable instead there is the possibility of missing data when the ExpectedVersion doesn't match PASI.

It is possible to pass zero as the ExpectedVersion all the time when calling the 'Get' services. In this scenario the most current object is always returned from PASI. The PASI client must check the version of each object returned to ensure it matched the version returned from IsDataAvailable. Without this check there is the possibility of data loss.

The PASI Core logically deletes records internally when required. To reduce data transfers between the PASI Core and the PASI clients, PASI will not return records that have been logically deleted. For example: if a student has three name records and one is "deleted" because it was created in error the next time the student is synchronized with the PASI Core, the student will contain two name records.

For types that are synchronization types (Student, Student School Enrolment, Course Enrolment, Section and Evaluated Mark) these values will be returned even if they are logically deleted. This gives the PASI client the ability to detect if a record has been logically deleted in the PASI core and the client can be updated accordingly. However, logically deleted synchronization types (Student, Student School Enrolment, Course Enrolment, Section and Evaluated Mark) should not be included for hash value calculation. For example, if a Student School Enrolment that has a registration status of "CreatedInError" (logically deleted), this enrolment and the enrolment status should not be included in the hash calculation. With this mechanism in place, a PASI Client is free from tracking synchronization types that are logically deleted.

In order to demonstrate how a client could use the PASI data synchronization functionality two reference clients have been created in VB.NET showing two possible approaches.

PASI supports full school year synchronization of Student School Enrolment, Section, Course Enrolment and Evaluated Mark for up to four school years. The calculation of which school years PASI is currently supporting full synchronization is based on the current date calendar year. Use the following to calculate the school years for which Student School Enrolment, Section, Course Enrolment and Evaluated Mark can be fully synchronized.